Last week I posted (in Off Topic) a problem I had where my player did a media check as part of an emplode re-synch, even though HiJack was installed with the media check option turned off, and the player had only been synched nine times previously on a new ("disk-buildered") single hard drive.

Paul Grzelak posted his best guess as to the problem, and Mark Lord concurred.

Well, I have a bit more information now, and while it appears that Paul was right, I don't have the solution to the problem.

It seems the media rebuild occurs any time I re-synch following an FTP file transfer. (When I use the player as a file taxi). Since other people manage to do this without a media check every time, I can only conclude I am doing something wrong. Below is how I load files into the player. (The unload procedure is identical, except for deleting the zip file, and of course using "get" instead of "put".)

=============================================

Batch file for loading the player:

@echo off
echo For this to work, you must have a directory on your PC
echo named '_FTP' and a file in that directory called 'empfile.zip'
echo containing the data you want to copy to your Player.
echo .
pause
cd C:\_FTP
ftp.exe -s:Taxi_In.txt
-------------------------------------------------------------------
ftp script for loading the player:

open 10.60.5.42


prompt
bin
hash
literal site rw
cd /drive0/var/taxi
delete empfile.zip
put empfile.zip
literal site ro
quit
================================================

It is acting very much as though the player were being left in rw mode after the ftp session, but I can't see why it would be.

The player is doing another media check as I type this, and I didn't run the ftp session through the batch/script files this time, but instead ran it manually, typing one line at a time from a DOS prompt. Every command line executed in ftp during this process seemed to return the appropriate response.

Any ideas? (And if it is because I'm doing something grossly stupid, maybe you could tell me via PM rather than putting my ignorance on display for the whole bbs?)

tanstaafl.

Question: Have you ever modified your default /bin/ro, rom, rw, rwm commands?

Reason: (And this one will be a trifle obscure, so be patient with me...) On the large disks, when you manually execute any of the mount commands that are default in the player distribution, there is a "mini-diskcheck" that is basically useless, but time consuming. You may note that doing a "rwm" command from the shell may take upwards of a minute (subjective time) to complete. This is because the command as written does not have the "nocheck" option built into it. For example:

rwm unmodified:
#!/bin/sh
mount -n -o remount,rw /drive0
[ -e /proc/ide/hdb ] && mount -n -o remount,rw /drive1

rwm modified:
#!/bin/sh
mount -n -o remount,rw,nocheck /drive0
[ -e /proc/ide/hdb ] && mount -n -o remount,rw,nocheck /drive1

I think I had this problem long, long ago, but I have been running with modified ro,rom,rw,rwm files for so long, I no longer remember. This may have fixed the problem.

Now, I do not know exactly how the ftp daemon / kernel implementation is changing the mount status, but it is certainly possible that your player is being powered off or quitting to shell before the change in mount status completes.

An interesting test would be to sit on the shell while all of this is happening. After your script completes, see if the partition is still read / write, if it is in the process of changing mount state or if it is read only.

Also, capture a boot after executing that batch. Capture also going into the shell and doing a "rw" and "rwm" command. Whatever file system is "dirty" will be flagged in the error messages on console, along with the text of the error.

He hasn't modified anything on the player as far as I know. I think he's using the 2.0 final player softwrae with Hijack, and that's all. The only modification he made was to create a "Taxi" folder under "Var" and occasionally copy files to it.

It might be worthwhile testing with a modified set of files to find out. Or, alternatively, forcing a delay at the end of the batch.

It might be worthwhile testing with a modified set of files to find out. Or, alternatively, forcing a delay at the end of the batch.


When I used ftp to put my "empfile.zip" into the player this afternoon, I didn't use the batch file at all.

I just manually typed things in, one line at a time, and got (AFAICT) appropriate responses to each command.

Obviously the rw command worked, because I was able to add the file. And it seemed like the ro command worked, in that there were no complaints displayed upon its execution. (I typed in: "ro" <rtrn> and got the response "OK".

So it doesn't seem to be a problem of the batch file or the script file over-running the command line or anything like that.

Modified set of files? That's 'way beyond my knowledge level... but when I type in the rw and ro commands, they execute instantly so does that mean they are working the way they should?

I don't see any sign of the player starting/stopping during this operation; in fact, I can do it while I'm listening to music and it never skips a beat. (It does seem to transfer the taxi file a bit slower when I do that). The player never shuts down or reboots.

Operationally, the player has been rock-solid, never a hint of a problem, which argues against some sort of hardware problem. I have added more than 60 GBs of music to the player in a single synch, with no trouble.

This whole thing has me puzzled, and its doubly frustrating because I don't know enough about ftp and linux to even know what to look for.

Oh, and as Tony said above, the player is completely stock and unmodified except for HiJack: a fresh single 80-GB disk-build with 2.0 final software on both the PC and the player. The only HiJack settings I have made are to disable the media check, and to specify a single disk drive.

tanstaafl.

The Hijack FTP "rw" and "ro" commands are self-contained -- they do not use the batch scripts on the player's filesystems (sorry Paul).

But they DO pop-up a message on the player's front panel display, indicating the "remounted" action taken (either "read-write" or "read-only").

Do you see those message(s) ??

Cheers

Do you see those message(s) ??

Yeah, Doug. What he said.

<glares at Doug>

(Was the first question I asked him in the instant messenger. He said he didn't know...)

The Hijack FTP "rw" and "ro" commands are self-contained -- they do not use the batch scripts on the player's filesystems

This is a bit confusing to me... I thought I ran the ftp.exe program from my PC, and thus the "ro" and "rw" commands would be from that executable. But I guess what you're saying is that the ftp.exe command just connects to the player and tells it what commands embedded within the player software are to be run?

Do you see those message(s) ??


I don't think to, but I wasn't looking for them so can't be sure.

It's too late for me to test this here at work today because I can't afford to be trapped here for another hour should I test it now and end up with a media check. (I don't allow myself to look at the bbs until after I have finished my work for the day or I'd never get anything done!)

I'll check it tonight when I get home and let you know what I see (or don't see).

tanstaafl.

I thought I ran the ftp.exe program from my PC, and thus the "ro" and "rw" commands would be from that executable.

Nope. Remember that you're not executing "ro" and "rw" commands. You're executing the LITERAL SITE command and telling it to send the strings "ro" or "rw" as its parameter. LITERAL SITE is the command to get the server to accept a command. In this case, the server is Hijack, so it's the one that executes the "ro" or "rw" command.

I'll check it tonight when I get home and let you know what I see (or don't see).


The "ro" and "rw" commands DO echo appropriately on the player, so that part is working correctly.

I used FTP to pull my "taxi" file off the player, then re-synched. It did a media check.

After an hour of media checking, I went to a DOS prompt and used FTP to open the player, then immediately closed it again without executing any commands. I re-synched, and got another media check.

After yet another hour of media checking, I went straight to emplode, and did a re-synch. This generated another media check. So apparently it wasn't the FTP session triggering the media checks.

So... today at work, I did the manual fsck as outlined in the FAQ:


To correct this error, enter the following at the shell prompt:

ro
umount /dev/hda4
<umount /dev/hdc4>
swapon /swapfile
fsck -fay /
fsck -fay /dev/hda4
<fsck -fay /dev/hdc4>
swapoff /swapfile
sync

(and then reboot the player by pulling its power cord)



(where the <lines> were skipped because I have a single drive system) and after an hour of media checking, I used FTP to delete the old taxi file, then went to emplode and did a re-synch.

It did NOT do a media check.

It appears that the fsck that emplode executes does not clear the "dirty" flag; only a manual fsck as shown above does that.

How the flag got set to "dirty" is a complete mystery to me. Up until the media-check problem began, I always ran my ftp sessions using batch/script files specifically to prevent forgetting to "ro" the filesystem.

tanstaafl.

Well, at least it's fixed now, and in a way that sort-of makes sense.

Did you see fsck produce any output about fixing things?

Greetings!

I am glad that it was something easy. Good work tracing it down and getting it fixed.

Something to watch for in the future - if you get the filesystem check in emplode, you know now (for a fact) that the emplode based check will not clear the error. Just go right in and do a manual check after the sync completes.

If you feel intensely bored and curious, you can always accidently leave a filesystem read/write, exit, try to sync with emplode while watching the terminal connection. Just to see if you see an error. Kind of boring, and will require two syncs (the failed emplode sync and a manual one) to restore your player to health. I would suspect a memory problem (or lack of memory, more explicitly).

Wish list item (bad location, I know): have emplode turn swap on / off for any disk checks it performs. Thanks.

Edit: Okay. I could not resist. The unfortunately amusing bit to all of this is that your first post in this thread had a link to a description of exactly this problem...

So you're saying that the reason the Emplode media check didn't fix the problem is because his drive was too big?

So you're saying that the reason the Emplode media check didn't fix the problem is because his drive was too big?

...and the amount of available memory too small. Yes. Or at least, that is what I suspect. In order to prove it, I would have to find out if the fsck that emplode performs can direct output over the serial port. I have not tried that particular step yet.

But I do know that if I perform a manual fsck on the 80GB drives from command line and the drives are almost full and I forget to turn on swap in advance, I will get an error from stage 5 of the fsck because it could not allocate enough space to complete the step.

Hm... It is a quiet morning. This may take some time, but I will try to get some output of this. It might be a good troubleshooting step, and perhaps good data for 3.0 enhancements. (Note: I am running 2.0final developer.)

Wish list item (bad location, I know): have emplode turn swap on / off for any disk checks it performs. Thanks.

It already does. If this is really the problem, it means your disk is so big that fsck is running out of swap if run when the player is loaded.

Peter

...Okay... That's a happy thought...

Do you know if there would be any serial output from an fsck performed via an emplode sync? I am grabbing the basic swap vs. no-swap comparison logs at the moment, but if you happen to remember off hand, that would save an experiment. Thanks!

Do you know if there would be any serial output from an fsck performed via an emplode sync? I am grabbing the basic swap vs. no-swap comparison logs at the moment, but if you happen to remember off hand, that would save an experiment. Thanks!

Fsck's stdout is swallowed by the player (it uses "fsck -C 1"), but its stderr, if any, should come out over serial.

Peter

Okay. So any error should go over serial. Cool.

Basically, this is the output I get when I try to perform a fsck on one of the music partitions of an 80GB drive without swap... This is the error (or something very similar) I suspect is happening when emplode's fsck does not clear the dirty flag...

empeg:/empeg/bin# fsck -fay /drive0

Parallelizing fsck version 1.19 (13-Jul-2000)

e2fsck 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

no room for private writable mapping

error: -12

fsck.ext2: Memory allocation failed while retrying to read bitmaps for /dev/hda4

empeg:/empeg/bin#

Okay. This was the hard bit. Forcing the partitions to be dirty. It just goes against everything I do normally...

I booted cleanly and then

<edited boot>

Prolux 4 empeg car - 2.1434 Mar 26 2003

Vcb: 0x4086d000

q

Restored terminal settings

Remounting first music partition read-only

Remounting second music partition read-only

Player exited normally: 0

Switching to shell-player loop

Starting bash.

empeg:/empeg/bin# rw

empeg:/empeg/bin# rwm

empeg:/empeg/bin# vi /waste.file

and yanked the power! Ow! But, it did what I wanted it to do...

<edited boot>

Prolux 4 empeg car - 2.1434 Mar 26 2003

Vcb: 0x4086d000

q

Restored terminal settings

Remounting first music partition read-only

Remounting second music partition read-only

Player exited normally: 0

Switching to shell-player loop

Starting bash.

empeg:/empeg/bin# rw

EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended

empeg:/empeg/bin# rwm

EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended

EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended

empeg:/empeg/bin# rom

empeg:/empeg/bin# ro

empeg:/empeg/bin# exit

Okay. Time for emplode, to see what I get from stdout... Rebooting just to make certain I start from the beginning, and I will touch a playlist to force a change...

Hmmm... That was quicker than expected...

Adding Swap: 16028k swap-space (priority -1)

e2fsck 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

VM: do_try_to_free_pages failed for player...

More to follow. Since it will try to check all of the partitions, this can take a long time... But we can already see it flailing around for memory...

Edit 1:

After about an hour, it finished its check on the /drive0 partition. When you do this a lot, you can actually hear the different stages going by, based on the change in disk activity. No additional errors appeared. Now we are on to /drive1...

e2fsck 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09

By the way, an interesting note: no sign of a check on the root partition... After the sync completes, I will exit emplode,
reboot the player and then go in manually and see exactly what the status of each partition is...

Greetings!

While this check is going on - a hypothesis. What if emplode checks all default root and music filesystems (/, /drive0, /drive1) for any errors, timers or mount counts. If it finds any problems, it schedules fsck runs. But what if emplode only runs fscks on the music partitions? Does emplode ever run an fsck against the root partition?

If this is the case, it is not a memory issue at all. Once the / partition gets dirty, emplode has no way of ever cleaning it. This would require manual intervention.

Just a theory...

It can fsck the root partition fine -- just like all other Linux installations on the planet! So nothing special there.

The reason I believe it is running out of memory -- actually it's probably running out of page frames -- is because the player software locks itself into memory, refusing to be paged out. This creates considerable pressure on physcial ram (aka. page frames), so much that fsck is unable to allocate the chunks that it requires to do its job. Maybe the player is neglecting to free() the music buffer space it normally nabs on startup?

Anyway, as observed, without a player running, fsck works just fine -- maybe even without the need for swap, though I haven't tested that.

Cheers

All of these theories can be tested by simply connecting to Hijack's webserver during the fsck, with and without the player running, and then using HTTP to examine /proc/slabinfo, /proc/meminfo, and perhaps a few other /proc entries..

Cheers

Does emplode ever run an fsck against the root partition?

No, but that isn't the problem here. Emplode runs fsck on each disk's music partition every time, and it's up to fsck whether it does a real check or whether it's satisfied that nothing dangerous has happened since last time. There's no way for dirtiness of / to affect fsck's decision about doing a full check of other partitions.

Peter

Greetings!

Hm... Okay. The emplode run did not clear the dirty flag on the root partition, but it did not force a resync on the next trip through.

<boot stuff>

Prolux 4 empeg car - 2.1434 Mar 26 2003

Vcb: 0x4086d000

q

Restored terminal settings

Remounting first music partition read-only

Remounting second music partition read-only

Player exited normally: 0

Switching to shell-player loop

Starting bash.

empeg:/empeg/bin# rw

EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended

empeg:/empeg/bin# rwm

empeg:/empeg/bin# rom

empeg:/empeg/bin# ro

empeg:/empeg/bin# exit

Note that the / partition is still dirty - it was not cleaned through emplode. But it did not force a disk check on resync. Hmmm....